{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make gender-balanced subset\n",
    "\n",
    "For most of the two centuries this project covers, more works of fiction were published by men than by women. This dataset experimentally levels that imbalance, using the simple technique of discarding works by men. In a few periods at the beginning and end of the timeline, more works were published by women, and we have to reverse the strategy.\n",
    "\n",
    "This approach means that coverage will be uneven across time; we'll have fewer works overall in periods where gender imbalance was severe.\n",
    "\n",
    "Given unlimited time, we could go back and select a supplement to fill in those dips. But I don't know for sure that this dataset will be used by anyone outside the current project; it seems unwise to be a perfectionist before knowing the size of the audience.\n",
    "\n",
    "The concept of \"gender balance\" implies a binaristic simplification of a more complex reality, but I want to avoid simplifying more than necessary. So I'm not going to discard works where gender is marked as 'u' or 'o.' Instead I'll downsample those works to keep them in proportion to the smaller subset.\n",
    "\n",
    "Mostly this is going to affect collections, anonymous works, and works with multiple authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "title = pd.read_csv('manual_title_subset.tsv', sep = '\\t', index_col = 'docid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START:  810 1526 394\n",
      "\n",
      "1800 20 27 7 \n",
      "1805 21 25 7 \n",
      "1810 34 19 7 *\n",
      "1815 23 28 9 \n",
      "1820 12 24 11 \n",
      "1825 13 30 9 \n",
      "1830 14 32 8 \n",
      "1835 19 40 8 \n",
      "1840 15 33 4 \n",
      "1845 11 23 13 \n",
      "1850 12 33 8 \n",
      "1855 19 28 5 \n",
      "1860 21 36 5 \n",
      "1865 28 26 3 *\n",
      "1870 19 28 9 \n",
      "1875 19 33 3 \n",
      "1880 19 25 4 \n",
      "1885 17 30 6 \n",
      "1890 24 34 0 \n",
      "1895 18 34 1 \n",
      "1900 11 37 5 \n",
      "1905 11 38 1 \n",
      "1910 11 39 2 \n",
      "1915 18 31 1 \n",
      "1920 15 34 2 \n",
      "1925 14 40 1 \n",
      "1930 14 39 4 \n",
      "1935 15 45 2 \n",
      "1940 17 31 6 \n",
      "1945 20 32 3 \n",
      "1950 20 35 3 \n",
      "1955 8 34 4 \n",
      "1960 14 36 1 \n",
      "1965 8 44 5 \n",
      "1970 13 34 5 \n",
      "1975 9 34 11 \n",
      "1980 15 37 8 \n",
      "1985 16 43 5 \n",
      "1990 18 29 13 \n",
      "1995 19 28 5 \n",
      "2000 14 41 5 \n",
      "2005 29 25 3 *\n",
      "\n",
      "(1501, 18)\n",
      "END:  686 686 129\n"
     ]
    }
   ],
   "source": [
    "numwomen = sum(title.gender == 'f')\n",
    "nummen = sum(title.gender == 'm')\n",
    "numother = len(title) - (numwomen + nummen)\n",
    "print(\"START: \", numwomen, nummen, numother)\n",
    "print()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for floor in range (1800, 2010, 5):\n",
    "    fiveyrs = title.loc[(title.firstpub >= floor) & \n",
    "                        (title.firstpub < floor + 5) &\n",
    "                        (title.category.isin({'longfiction', 'shortfiction'}))\n",
    "                        , :]\n",
    "    \n",
    "    numwomen = sum(fiveyrs.gender == 'f')\n",
    "    nummen = sum(fiveyrs.gender == 'm')\n",
    "    numother = len(fiveyrs) - (numwomen + nummen)\n",
    "    if numwomen > nummen:\n",
    "        asterisk = \"*\"\n",
    "    else:\n",
    "        asterisk = ''\n",
    "        \n",
    "    print(floor, numwomen, nummen, numother, asterisk)\n",
    "    \n",
    "    women = fiveyrs.loc[fiveyrs.gender == 'f', :]\n",
    "    men = fiveyrs.loc[fiveyrs.gender == 'm', :]\n",
    "    others = fiveyrs.loc[(fiveyrs.gender == 'u') | (fiveyrs.gender == 'o'), :]\n",
    "    \n",
    "    if numwomen < nummen:\n",
    "        men = men.sample(n = numwomen)\n",
    "    elif nummen < numwomen:\n",
    "        women = women.sample(n = nummen)\n",
    "    \n",
    "    reduction = (len(men) + len(women)) / (nummen + numwomen)\n",
    "    numothers = len(others)\n",
    "    reduced = int(numothers * reduction)\n",
    "    \n",
    "    if reduced > 0:\n",
    "        others = others.sample(n = reduced)\n",
    "        newframe = pd.concat([women, men, others], sort = False)\n",
    "    else:\n",
    "        newframe = pd.concat([women, men], sort = False)\n",
    "        \n",
    "    frames.append(newframe)\n",
    "    \n",
    "print()\n",
    "newtitle = pd.concat(frames)\n",
    "print(newtitle.shape)\n",
    "numwomen = sum(newtitle.gender == 'f')\n",
    "nummen = sum(newtitle.gender == 'm')\n",
    "numother = len(newtitle) - (numwomen + nummen)\n",
    "print(\"END: \", numwomen, nummen, numother)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "newtitle.to_csv('gender_balanced_subset.tsv', sep = '\\t', index_label = 'docid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
